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ABSTRACT 

The  North  Atlantic  Treaty  Organization  (NATO)  Research  and  Technology  Organization  (RTO)  organized 
and  conducted  a  workshop  on  human  factors  and  performance  issues  related  to  embedded  virtual  simulation 
(EVS).  The  workshop  was  held  on  20  -  22  October  20,  2009  at  the  University  of  Central  Florida  -  Institute 
for  Simulation  and  Training  (UCF  -  1ST)  in  Orlando,  Florida.  The  purpose  was  to  address  human 
effectiveness  issues  related  to  application  of  EVS  to  the  military  domain.  Papers  were  presented  describing 
the  current  state  of  EVS  technology.  The  conference  participants  generally  agreed  that  technological  and 
conceptual  advances  have  led  to  the  increased  utility  and  effectiveness  of  EVS.  At  the  same  time,  participants 
also  noted  that  the  technology  has  not  been  generally  accepted  by  system  arid  training  developers.  Reasons 
for  the  resistance  to  adopt  EVS  were  identified,  and  a  possible  way  forward  was  discussed. 


1.0  INTRODUCTION 

The  North  Atlantic  Treaty  Organization  (NATO)  Research  and  Technology  Organization  (RTO)  organized 
and  conducted  a  workshop  on  human  factors  and  performance  issues  related  to  embedded  virtual  simulation 
(EVS).  The  workshop  was  held  on  20  -  22  October  20,  2009  at  the  University  of  Central  Florida  -  Institute 
for  Simulation  and  Training  (UCF  -  1ST)  in  Orlando,  Florida. 

1.1  Background 

In  its  “Call  for  Papers,”  the  NATO  RTO  identified  the  need  for  units  to  deploy  with  little  or  no  notice  and  to 
adapt  to  evolving  situations.  This  places  units  in  locations  where  they  do  not  have  the  facilities  and 
infrastructure  needed  to  train  for,  optimally  plan,  and  rehearse  complex  missions.  The  use  of  EVS  is  seen  as 
an  essential  tool  to  provide  more  effective  deployed  training.  EVS  is  a  concept  that  tightly  integrates  training 
and  mission  functionality  into  operational  equipment.  Recent  advances  in  training  concepts,  intelligent  agent 
technologies,  computers,  communication,  and  display  technologies  offer  new  opportunities  for  embedded 
training  and  mission  preparation/rehearsal  capabilities  in  a  highly  mobile  military. 

Although  embedded  virtual  simulation  is  a  known  concept  with  many  advantages,  it  has  been  rarely 
implemented  in  air  and  land  platforms.  The  most  challenging  applications  are  deployed  settings  that  provide 
little  support  to  users  in  the  way  of  instructional  staff  and/or  infrastructure.  To  meet  this  challenge,  embedded 
simulation  will  have  to  include  a  range  of  capabilities,  e.g.,  an  intelligent  tutor,  beyond  those  of  existing 
simulation  centres  and  simulators.  Integrating  Virtual  Reality  (VR),  Augmented  Reality  (AR),  and  intelligent 
agent  technologies  in  novel  ways  can  significantly  enlarge  the  area  of  application  and  introduce  new 


RTO-MP-HFM-1 69 


T- 1 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

OCT  2009 


2.  REPORT  TYPE 

N/A 


3.  DATES  COVERED 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


4.  TITLE  AND  SUBTITLE 

Technical  Evaluation  Report:  Human  Dimensions  In  Embedded  Virtual 
Simulation 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  8.  PERFORMING  ORGANIZATION 

Institute  for  Defense  Analyses  4850  Mark  Center  Drive  Alexandria,  report  number 

Virginia  22311-1882 

9.  SPONSORING/MONITORING  AGENCY  NAME(S )  AND  ADDRESS(ES )  10.  SPONSOR/MONITOR' S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADA562526.  RTO-MP-HFM-169  Human  Dimensions  in  Embedded  Virtual  Simulation  (Les 
dimensions  humaines  dans  la  simulation  virtuelle  integree),  The  original  document  contains  color  images. 

14.  ABSTRACT 

The  North  Atlantic  Treaty  Organization  (NATO)  Research  and  Technology  Organization  (RTO) 
organized  and  conducted  a  workshop  on  human  factors  and  performance  issues  related  to  embedded 
virtual  simulation  (EVS).  The  workshop  was  held  on  20  22  October  20,  2009  at  the  University  of  Central 
Florida  Institute  for  Simulation  and  Training  (UCF  1ST)  in  Orlando,  Florida.  The  purpose  was  to  address 
human  effectiveness  issues  related  to  application  of  EVS  to  the  military  domain.  Papers  were  presented 
describing  the  current  state  of  EVS  technology.  The  conference  participants  generally  agreed  that 
technological  and  conceptual  advances  have  led  to  the  increased  utility  and  effectiveness  of  EVS.  At  the 
same  time,  participants  also  noted  that  the  technology  has  not  been  generally  accepted  by  system  and 
training  developers.  Reasons  for  the  resistance  to  adopt  EVS  were  identified,  and  a  possible  way  forward 
was  discussed. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


17.  LIMITATION  OF 

18.  NUMBER 

ABSTRACT 

OF  PAGES 

SAR 

12 

19a.  NAME  OF 
RESPONSIBLE  PERSON 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Technical  Evaluation  Report 


e-*a‘r!.A-<n‘vi  Artt/rtkr 


capabilities  that  would  enhance  the  systems’  effectiveness.  By  utilizing  network-enabled  capabilities,  it  is 
possible  to  provide  team,  collective,  joint,  and  coalition  training  and  mission  preparation/rehearsal  on  a  wide 
variety  of  military  skills,  including  warfighting,  peace  keeping  and  maintenance. 

The  military  needs  to  understand  the  risks  and  benefits  of  embedded  virtual  simulation.  It  is  important  to 
carefully  consider  the  science  that  can  produce  new  embedded  virtual  simulation  strategies  and  technologies. 
It  is  also  important  to  understand  how  individual  performance  is  changed  by  the  use  of  embedded  virtual 
simulation. 

Based  on  this  need,  NATO  RTO  issued  a  call  for  papers  to  be  presented  and  discussed  at  the  workshop.  In 
addition  to  the  papers,  the  workshop  committee  members  conducted  a  series  of  mind  mapping  exercises 
designed  to  brainstorm  concepts  related  to  EVS:  requirements  identification,  training  management,  interface 
technology,  and  learning.  A  detailed  agenda  of  workshop  papers  and  activities  is  provided  in  the  Appendix. 

1.2  Workshop  Goals  and  Objectives 

In  their  introductory  presentations  to  the  workshop,  Dr.  Robert  Sottilare  (USA)  and  Dr.  Thomas  Alexander 
(DEU)  and  set  the  workshop  goals  and  objectives.  Dr.  Alexander  described  the  workshop’s  primary  goal  was 
“...to  address  human  effectiveness  issues  across  conceptual,  functional  and  technological  levels  when  using 
embedded  virtual  simulation  in  the  military  domain.”  Dr.  Robert  Sottilare  (USA)  identified  the  specific 
workshop  objectives  as  follows: 

•  Military  requirements  for  embedded  simulation  including  policy  and  user  requirements 

•  Operational  constraints  of  embedded  simulation 

•  Training  and  mission  support  requirements  including  intelligent  tutoring,  training  management, 
performance  measurement  and  feedback  (e.g.,  after-action  review  in  embedded  simulation) 

•  Human -Technology  Interfaces 

•  Application  of  intelligent  agents 

•  Current  and  future  directions  for  virtual  simulation  in  operational  platforms 

2.0  KEYNOTE  ADDRESSES 

Following  the  introductory  presentations,  four  invited  speakers  were  asked  to  describe  the  current  state  of  the 
art  in  EVS  with  examples  from  differing  applications.  The  first  of  the  keynote  addresses  was  presented  by  Dr. 
Lochlan  Magee  (CAN),  who  described  a  number  of  applications  to  NATO  militaries  that  illustrate  recent 
progress  in  EVS  technologies.  Nevertheless,  embedded  training  continues  to  pose  significant  problems  for 
general  application  to  military  problems.  The  challenges  he  posed  to  the  workshop  were  to  identify  gaps  and 
to  find  potential  solutions  that  will  allow  effective  implementation  of  embedded  virtual  simulation. 

Lt.  Col.  Gerbe  G.  Verhaaf  of  the  Royal  Netherlands  Air  Force  Command  discussed  EVS  technology  from  the 
user’s  perspective.  He  maintained  that  the  requirement  to  fight  asymmetric  threats  while  deployed  in  a 
multinational  coalition  demands  a  training  method  that  promotes  high  levels  of  proficiency  but  is  less 
dependent  on  traditional  instrumented  ranges.  As  an  example  of  such  an  advanced  method,  he  described  test 
versions  of  the  F-16  MLU  (“Orange  Jumper”).  This  demonstration  aircraft  employed  an  EVS  systems  that 
allow  training  on  the  entire  live  and  kill  chain.  This  test  bed  serves  as  a  prototype  for  the  F-35  Joint  Strike 
Fighter,  for  which  embedded  training  is  a  design  requirement. 
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Dr.  Dirk  Schmidt  (DEU)  of  Krauss-Maffei  Wegmann  reviewed  EVS  from  the  research  perspective.  He 
presented  his  company’s  research  into  EVS  technology  for  the  Infantry  Fighting  Vehicle  (IFV)  Puma.  His 
research  findings  indicated  that  embedded  training  systems  can  be  successfully  incorporated  in  ground  combat 
vehicles.  However,  he  noted  that,  for  embedded  training  on  a  single  vehicle,  the  vehicle  commander  needs  to 
be  trained  as  instructor  and  needs  support  by  a  Learning  Management  System  (LMS).  Dr.  Schmidt  also  argued 
that,  in  addition  to  crew  training,  an  EVS  potentially  provides  a  technological  basis  for  mission  rehearsal  and 
decision  support  during  missions. 

Mr.  James  A.  Shiflett  (USA)  of  Science  Applications  International  Corporation  (SAIC)  discussed  EVS  from  a 
developer’s  perspective.  His  example  was  the  Future  Combat  System  (FCS),  which  was  recently  significantly 
restructured  and  no  longer  includes  the  development  of  manned  ground  vehicles.  In  its  original  concept,  the 
FCS  vehicle  development  program  included  embedded  training  as  a  key  performance  parameter  (KPP)  in 
system  development.  (KPPs  are  those  system  attributes  considered  most  critical  or  essential  for  an  effective 
military  capability.)  The  FCS  experience  demonstrated  that  embedded  training  is  a  technically  feasible  and 
mature  application  to  any  ground  vehicle.  Cost  estimates  for  building  embedded  training  into  FCS  was  about 
3%  of  total  system  costs.  Despite  the  redirection  of  FCS,  embedded  training  remains  a  KPP  for  any  new  Army 
combat  vehicle.  Mr.  Shiflett  further  argued  that  embedded  training  can  be  incorporated  into  existing  systems  if 
they  are  linked  by  a  system-of-systems  common  operating  environment  (SOS  COE)  architecture  and  provide 
appropriate  warrior-machine  interfaces  (WMIs). 


3.0  SESSION  1:  POLICY  AND  USER  REQUIREMENTS  FOR  EMBEDDED 
TRAINING 

Session  1  was  led  by  Dr.  Stephen  Goldberg  (USA)  of  the  US  Army  Research  Institute  (ARI).  This  session 
identified  and  defined  policy  and  user  requirements  for  EVS  technology  through  the  examination  of  lessons 
learned  from  specific  embedded  training  applications.  By  way  of  introduction,  he  reminded  the  workshop  that 
policy  toward  embedded  training  dates  back  to  1987  when  GEN  Max  Thurman,  commander  of  TRADOC, 
wrote  an  instruction  that  embedded  training  was  considered  the  first  option  for  any  newly  acquired  operational 
system.  Although  the  policy  was  clear,  it  lacked  appropriate  enforcement  provisions  and  was,  for  the  most 
part,  effectively  ignored. 

3.1  Presentations  in  Session  1 

Brian  Crabb,  Jennifer  Phillips,  and  William  Ross  asserted  that  the  future  Network  Centric  Environment  (NCE) 
adds  substantial  lethality  and  effectiveness  to  ground  units.  At  the  same  time,  such  increases  in  technical 
capability  also  add  to  the  cognitive  complexity  of  unit  leader  tasks  and  skills.  Crabb  et  al.  used  the  Dreyfus 
and  Dreyfus  (1986)  model  of  expert  development  to  describe  how  future  leaders  could  acquire  the  necessary 
levels  of  competence  required  by  the  NCE.  Such  models  typically  specify  that  expertise  is  dependent  on  long 
periods  (>  10  years)  of  deliberate  practice.  They  proposed  that  embedded  training  technology  could  possibly 
break  the  10-year  rule  by  providing  high  levels  of  repetitive  practice  with  automated  guidance  on  a  wide 
variety  of  conditions. 

Bills,  Flachsbart,  Olson,  and  Kern  discussed  applications  of  embedded  training  in  the  F-35  (Lightning  II), 
previously  referred  to  as  the  Joint  Strike  Fighter.  The  training  system  is  in  this  aircraft  is  fully  embedded  in 
the  aircraft’s  integrated  core  processer  (ICP),  but  partitioned  into  two  components.  One  component  is  the 
virtual  training  (VT)  system,  which  represents  an  extension  of  the  system  developed  by  Royal  Netherlands  Air 
Force  for  the  F-16.  The  VT  system  enhances  the  training  environment  by  overlaying  constructive  simulation 
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on  the  real  world.  The  other  component  is  the  P5  Combat  Training  System  that  is  integrated  with  the  air 
combat  manoeuvring  instrumentation  (ACMI)  to  provide  on-board  “rangeless”  live  combat  training 
capability.  Cost  models  of  the  F-35  program  have  projected  cost-avoidance  savings  of  nearly  $3B  USD  over 
the  full  life  cycle  of  the  aircraft. 

3.2  Mind  Mapping  Exercise  for  Session  1 

The  first  mind  mapping  exercise  was  facilitated  by  Drs.  Goldberg  and  Magee  and  sought  to  identify  EVS 
requirements.  The  following  describes  the  three  specific  questions  asked,  and  some  of  the  answers  provided 
through  the  exercise: 

3.2.1  What  potential  operational  systems  are  good  candidates  for  application  of  EVS? 

The  breakout  groups  provided  examples  of  EVSs  for  a  variety  of  operational  systems.  However,  some 
examples  had  more  challenges  than  others.  For  instance,  C4ISR  systems  have  computation  power  that  could 
be  used  to  realistically  stimulate/simulate  EVS  targets.  On  the  other  hand,  EVS  systems  on  fast  moving 
aircraft  pose  challenges  in  space  requirements  and  simulation  fidelity. 

3.2.2  What  are  the  human  factors  challenges  for  training  and  performance? 

The  blending  of  live  and  virtual  simulation  poses  fidelity  as  well  as  safety  issues.  The  primary  pedagogical 
challenge  is  how  to  incorporate  secondary  roles  (e.g.,  battle  master,  scenario  generators,  and  role  players)  in  a 
constrained  operational  environment. 

3.2.3  What  are  the  limitations  of  applying  EVS,  particularly  those  that  have  hampered 
implementation  of  the  technology? 

Concerns  include  space  requirements,  wear-and-tear  on  operational  equipment,  interference  with  operational 
equipment/effectiveness,  and  limited  realism  (particularly  for  close-in  targets). 

4.0  SPECIAL  SESSION:  STRESS  AND  SUICIDAL  BEHAVIOR 

One  of  the  hallmarks  of  embedded  training  is  that  it  is  designed  to  be  used  in  the  deployed  environment.  In 
this  special  session,  Pregelj  pointed  out  that  this  environment  has  considerable  negative  emotional  or 
behavioural  consequences  that  could  be  associated  with  higher  suicide  risk.  Research  conducted  in  Slovenia 
on  psychiatric  populations  indicated  that  patients  with  stress-related  disorders  expressed  suicidal  thoughts  and 
attempted  suicide  more  often  than  patients  with  non-stress-related  problems.  Comparisons  with  suicide 
victims  having  no  previous  histories  of  aggressive  behaviours,  victims  with  aggressive  pasts  were  more  likely 
to  have  more  negative  life  events  connected  with  private  life  in  a  month  before  suicide,  to  have  attempted 
suicide  more  often,  and  to  be  successful  in  first  suicide  attempts.  Preglj  also  presented  possible  physiological 
mechanisms  for  the  link  between  stress  and  suicide  that  involved  dysfunctions  of  the  serotoninergic 
neurotransmitter  system. 
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5.0  SESSION  2:  HUMAN  EFFECTIVENESS  AND  EMBEDDED  TRAINING 
ENVIRONMENTS 

Session  2  was  facilitated  by  Dr.  Dee  Andrews  (USA)  with  the  US  Air  Force  Research  Laboratory  (AFRL)  and 
by  Dr.  Jan  Roessingh  (NED)  with  the  Dutch  National  Aerospace  Laboratory.  Topics  included  the 
management  of  embedded  training  and  distributed  modes  for  embedded  training. 

5.1  Presentations  in  Session  2 

Roessingh  and  Verhaaf  explored  the  training  effectiveness  of  embedded  training  in  a  multi-fighter 
environment.  They  cited  research  showing  gains  of  30%  in  effectiveness  at  the  same  cost  by  using  embedded 
training  in  fighter  aircraft.  They  also  considered  more  qualitative  benefits.  For  instance,  they  explored  how 
embedded  training  relates  to  user  needs  and  how  embedded  training  could  best  contribute  to  the  fulfilment  of 
the  training  objectives  for  fighter  pilots.  They  also  provided  a  qualitative  identification  of  the  training 
effectiveness  of  embedded  training  when  compared  with  “legacy”  live  training  exercises  or  tactical  training  on 
simulators.  Finally,  the  integration  of  embedded  training  with  other  forms  or  live,  virtual,  and  constructive 
training  was  examined. 

Cannon-Bowers  reviewed  the  application  of  embedded  training  for  high  performance  teams.  The  prototypical 
decision  environment  for  her  discussion  was  the  AEGIS  Combat  System,  which  involves  highly  complex 
team  interactions  under  time  pressure.  Her  experience  with  such  systems  indicated  that  there  are  two  essential 
questions:  (1)  How  do  experts  perform  their  tasks?  (2)  How  do  we  turn  a  team  of  experts  into  an  expert  team? 
The  implications  of  these  questions  have  been,  first,  to  create  realistic  simulations  that  provide  synthetic 
experience  to  accelerate  the  development  of  individual  expertise.  Second,  these  simulations  must  immerse 
teams  in  realistic,  multi-player  scenarios  that  build  shared  mental  models,  foster  self-correction,  enhance 
explicit  coordination,  and  promote  implicit  communication.  The  challenge  of  embedded  training  is  to 
implement  scenario-based  training  with  minimal  external  support.  Intelligent  tutoring  systems  is  a  possible 
approach  to  automating  functions  such  as  scenario  generation,  data  collection  and  interpretation,  diagnosis  of 
performance  deficiencies,  and  provision  of  feedback. 

Endsley,  Riley,  and  Strater  viewed  embedded  training  as  an  appropriate  platform  for  enhancing  high-level 
cognitive  skills,  including  situation  awareness  (SA).  They  described  a  set  of  principled  approaches  for  training 
the  cognitive  skills  that  underlie  situation  awareness  (SA)  and  decision  making  that  can  be  easily  employed  in 
future  embedded  training  systems  and  can  enhance  their  capabilities.  These  training  approaches  are  (1)  the 
SA  Trainer,  which  employs  systematic  experiential-based  training  to  build  essential  knowledge  bases  and 
higher-order  cognitive  skills;  (2)  the  Virtual  Environment  Situation  Awareness  Review  System  (VESARS), 
which  employs  informed  feedback  on  SA  skills,  team  communications,  and  the  accuracy  and  completeness  of 
SA;  and  (3)  the  SA  Virtual  Instructor  (SAVI),  which  builds  critical  SA  skills  by  calibration  of  good  and  poor 
SA  practices.  The  purpose  of  these  tools  is  to  develop  and  hone  the  needed  SA  skills  for  achieving  high  levels 
of  performance. 
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5.2  Mind  Mapping  Exercises  for  Session  2 

Dr.  Andrews  and  Dr.  Roessingh  moderated  the  second  round  of  mind  mapping  exercises,  which  focused  on 
the  following  four  questions:1 

5.2.1  What  are  viable  instructional  strategies  for  EVS? 

Although  a  wide  variety  of  strategies  may  apply,  EVS  is  best  used  in  deployed  (vice  school  house)  situations. 
Different  aspects  of  deployed  training  apply,  such  as  reachback,  cross-training,  and  refresher/sustainment 
training. 

5.2.2  What  categories  of  metrics  can  be  use  to  manage  EVS? 

Again,  the  full  range  of  metrics  applies  to  EVS,  from  behavioural  process  measures  (including 
psychophysiological  indicators,  such  as  eye  movements)  to  tasks  outcomes.  However,  the  applicability  of 
qualitative  judgments  (i.e.,  “style”  points)  may  have  limited  applicability  to  the  extent  that  they  require 
additional  experts  outside  of  team/crewmembers. 

5.2.3  How  do  you  manage  training  sessions? 

The  principal  question  in  management  of  EVS  is  determining  the  trainer.  Can  a  leader  or  other  team  member 
also  assume  the  role  of  trainer?  Implementation  of  intelligent  tutoring  systems  may  help,  at  least  in  part.  Also, 
an  EVS  must  incorporate  aspects  of  learning  management  systems  (LMSs),  such  as  trainee  profiles.  Such 
profiles  would  contain  data  related  to  experience,  learning  styles,  and  so  forth.  Prototypical  LMSs  are 
implemented  in  Sweden  (Saab  system)  and  Canada. 

5.2.4  How  do  you  envision  tasks  currently  accomplished  by  “white  cell”  personnel? 

In  order  to  be  implemented  in  EVSs,  control  functions  normally  performed  by  white  cells  must  be  automated 
to  the  extent  possible.  However,  there  is  still  a  role  for  human  white  cell  participants  to  provide  instruction 
aimed  at  higher  echelons  or  to  provide  reachback  capabilities. 


6.0  SESSION  3:  HUMAN  INTERACTION  WITH  EMBEDDED  VIRTUAL 
SIMULATIONS 

Dr.  Thomas  Alexander  (DEU)  and  Dr.  Sylvain  Hourlier  (FRA)  moderated  the  third  session,  which  focused  on 
interface  technology  including  sensors  and  controls.  The  effectiveness  of  such  interfaces  was  discussed  with 
respect  to  visual  perception  and  decision  making  processes,  such  as  target  detection  and  identification. 

6.1  Presentations  in  Session  3 

Neuhofer  discussed  a  training  system  that  embedded  augmented  reality  for  a  dynamic  industrial  task  requiring 
human-robot  cooperation.  The  task  was  to  remove  undesired  sand  relics  on  cast  metal  parts  using  deep  frozen 
CO2  as  residue-free  detergent.  In  this  task,  the  robot  performs  the  heavyweight  part  handling  whereas  the 
human  focuses  on  control,  relics  removal,  and  inspection.  An  immersive  simulation  was  used  to  emulate  this 
industrial  environment  and  to  test  two  control  configurations.  The  augmented  reality  (AR)  provided  an  optical 

1  A  fifth  question  was  able  posed:  How  will  you  manage  training  in  different  modes  (home  station,  in  transit,  in  theatre)?  None  of 
the  breakout  groups  was  able  to  answer  this  question  within  the  time  allotted. 
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see-through  display  where  the  user  sees  the  real  control  tool  in  his  hand.  This  configuration  was  compared  to  a 
pure  virtual  reality  (VR)  mode  where  object  visualization  was  artificial.  Comparison  of  performance  in 
execution  times  and  shooting  accuracy  indicated  no  differences  between  AR  and  VR.  NASA  TLX  surveys 
indicated  relatively  low  mental  and  physical  loads  for  both  conditions,  but  there  was  a  clear  personal 
preference  for  the  AR  configuration,  especially  among  women. 

Embedded  training  has  the  capability  to  combine  virtual  or  augmented  input  with  real  inputs  from  aircraft 
cockpits.  Sandor  et  al.  explored  the  possible  deleterious  effects  of  mismatches  between  synthetic  inputs  from 
embedded  training  systems  and  real  inputs.  Their  first  experiment  examined  the  effects  of  visual  structure  in  a 
virtually  tilted  environment.  Results  indicated  that  greater  visual  structure  induced  greater  perceived  distortion 
in  tilt.  Also,  purely  virtual  controls  induced  greater  distortion  than  did  a  combined  virtual-kinetic  control.  The 
second  experiment  looked  at  the  perceiver’s  frame  of  reference  on  the  fusion  of  auditory  with  visual  inputs. 
Previous  research  indicated  that  visual-auditory  fusion  is  most  affected  by  disparities  in  azimuth.  Findings 
from  the  present  research  indicated  that  fusion  was  also  affected  by  an  integration  of  egocentric  (bodily 
orientation)  and  allocentric  (external  visual)  cues. 

Dyer  reported  on  the  application  of  embedded  training  concepts  in  the  Ground  Soldier  System  (GSS) 
prototypes  that  are  currently  in  development.  The  major  components  of  these  prototypes  are  a  wearable 
computer  and  a  global  positioning  system.  The  soldier  receives  input  from  this  system  via  helmet-mounted  or 
goggle-mounted  display  and  manipulates  the  input  via  a  soldier/system  control  unit.  The  GSS  prototypes  were 
examined  for  their  capability  to  provide  training  in  two  areas.  The  first  is  individual  instruction  on  a  subset  of 
soldier  tasks,  which  were  selected  to  meet  both  psychological  criteria  (e.g.,  hard  to  perform,  infrequently 
performed)  and  warfighting  criteria  (e.g.,  lethality  and  survivability).  The  second  area  is  collective  training  in 
a  virtual  environment.  The  current  collective  concept  under  examination  was  to  integrate  the  GSS  operational 
system  in  an  external  virtual  environment.  The  conclusion  of  initial  testing  was  that  the  potential  for  EVS 
training  exists  for  GSSs,  but  its  application  is  limited. 

6.2  Mind  mapping  Exercises  for  Session  3 

Dr.  Alexander  and  Dr.  Hourlier  moderated  the  third  mind  mapping  exercise,  which  addressed  a  single 
question:  What  are  the  limitations  of  embedded  virtual  simulation  interface  technology?  It  was  generally 
noted  that  dismounted  soldier  systems,  such  as  described  by  Dr.  Dyer,  present  the  most  difficult 
implementation  problems  for  EVS.  More  specific  conclusions  were  discussed  with  respect  to  the  following 
interface  sensory  modalities: 

6.2.1  Vision 

With  respect  to  all  modalities,  visual  displays  are  the  best  developed  capabilities.  Whereas  visual  cues  are  best 
when  simulating  day  vision  with  the  unaided  eye,  difficulties  exist  for  night  vision.  Also,  visual  displays  may 
be  problematic  for  tasks  that  depend  on  depth  cues  and  field  of  view. 

6.2.2  Sound 

Impressive  Hollywood  sound  effects  have  been  demonstrated.  Localization  effects  are  more  difficult  to 
simulate.  Such  problems  and  individual  differences  in  sensitivity  may  be  mitigated  by  the  fact  that  many 
performers  wear  headphones  in  operational  environment,  which  would  also  be  employed  in  EVS. 
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6.2.3  Motion  and  Vibration 

The  most  advanced  work  on  motion  cuing  has  been  done  in  aircraft  research.  This  research  indicates  that  full 
motion  simulation  is  expensive  and  its  effectiveness  is  questionable.  Lower  cost  motion  simulations  (haptic 
belts  and  vests)  provide  cheaper  and  potentially  more  effective  alternative. 

6.2.4  Chemical  Senses 

Devices  exist  to  simulate  certain  odours,  but  problems  pertain  to  dissipation  in  environment.  Except  for  some 
medical  tasks,  it  is  also  questionable  whether  there  are  many  tasks  that  are  cued  or  regulated  by  smells. 


7.0  SESSION  4:  LEARNING  AND  HUMAN  EFFECTIVENESS  IN  EMBEDDED 
VIRTUAL  SIMULATIONS 

Ms.  Claudy  Koerhuis  (NED)  acted  as  facilitators  for  the  fourth  session,  which  focused  on  topics  in  student 
modeling,  coaching  strategies,  and  tutor  perception. 

7.1  Presentations  in  Session  4 

Dr.  Robert  Sottilare  (USA)  made  the  argument  that  one  of  the  ways  that  a  human  tutor  adjusts  one-on-one 
instruction  is  on  the  basis  of  the  perceived  affective  state  of  the  learner.  Adjustments  to  instructional  strategy 
can  alter  those  moods,  which  in  turn  impacts  the  effectiveness  of  learning.  Some  initial  research  findings 
indicate  that  there  are  computer  algorithms  for  detecting  affective  states  that  could  be  applied  to  intelligence 
tutoring  systems.  To  be  implemented  in  embedded  training  such  algorithms  should  be  accurate,  portable, 
affordable,  passive,  and  effective.  Given  deficiencies  in  these  areas,  Sottilare  recommended  three  areas  for 
further  research:  sensors,  predictive  models,  and  building  trust. 

Jensen,  Mosley,  Sanders,  and  Sims  explored  the  efficacy  of  feedback  methods  for  intelligent  embedded 
tutoring  systems.  Their  paper  reviewed  two  case  studies  of  embedded  training  designed  to  deliver  feedback 
with  little  or  no  instructor  involvement.  The  first  was  a  man-wearable  trainer  for  dismounted  operations, 
which  provided  feedback  on  the  individual  soldier  position  and  tracking  behaviour,  and  presented  semi- 
automated  force  (SAF)  entities  to  simulate  other  soldiers.  Feedback  from  soldiers  indicated  that  most  effective 
training  was  provided  for  simple  procedural  tasks  (maintain  muzzle  awareness  and  maintain  sectors  of  fire), 
whereas  the  least  effective  training  were  those  requiring  coordination  with  SAF  entities  (team  movement  and 
room  clearing.  The  second  case  study  involved  a  robotic  vehicle  control  station  trainer,  and  compared  two 
feedback  conditions:  Immediate  Directive  Feedback  (IDF)  provided  by  the  intelligent  tutoring  system  versus 
human-facilitated  after  action  reviews  (AAR)s  using  open-ended,  content  neutral  prompts.  The  relative 
effectiveness  of  these  two  approaches  depended  on  type  of  tasks:  Results  indicated  decreased  errors  for  IDF 
condition  on  procedural  tasks,  but  increased  retention  for  AARs  on  conceptual  tasks. 

Bell  and  Short  noted  that  communication  and  team  coordination  have  been  overlooked  in  embedded 
simulation  because  complex  and  highly  verbal  interactions  are  difficult  to  simulate  without  human  role 
players.  The  capability  to  train  communication  and  coordination  was  seen  as  particularly  relevant  to  deployed 
NATO  units.  To  address  this  issue,  the  authors  employed  a  cognitive  modelling  tool  that  encapsulates  human 
expertise  and  behaviour  in  synthetic  agents.  These  agents  incorporate  speech  recognition,  conversation 
management,  and  speech  generation  capabilities.  These  tools  were  implemented  in  synthetic  environments  for 
training  two  different  domains.  One  domain  was  air-ground  communications  task  between  pilot  and  the  tower 
controller  wherein  the  role  of  the  controller,  as  well  as  the  instructor  pilot,  was  simulated.  Use  of  this  system 
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was  positively  correlated  with  reduced  time  to  achieve  a  rating  of  “good”  on  flown  sorties.  The  other  domain 
was  mission  planning  and  rehearsal  for  close  air  support  (CAS).  The  same  system  was  embedded  in  the 
Combined  Arms  Gateway  Environment  (CAGE)  which  is  design  to  plan,  rehearse,  and  conduct  missions. 
Within  a  CAS  scenario,  synthetic  agents  were  used  to  role  play  the  Joint  Terminal  Attack  Controller  (JTAC) 
and  interact  with  a  pilot  user.  An  operational  evaluation  of  this  system  showed  that  voluntary  use  was 
positively  correlated  with  training  gains  in  situation  awareness,  communications,  and  in-flight  checks. 

To  minimize  the  need  for  instructional  staff  or  infrastructure,  Heuvelink,  van  den  Bosch,  van  Doesburg,  and 
Harbers  advocated  the  use  intelligent  agents  that  emulate  the  actions  of  fellow  team  members,  opponents,  and 
instructors.  The  thesis  of  their  paper  was  that  there  is  also  a  need  for  a  director  agent  (DA)  to  diagnose  the 
trainee’s  task  performance,  instruct  agents  to  perform  certain  behaviour,  and  steer  the  simulation.  The  goal  of 
the  DA’s  actions  is  to  tailor  the  training  scenario  so  that  it  is  a  bit  more  difficult  than  the  trainee’s  current  level 
of  performance.  This  concept  was  tested  on  a  single -player  table -top  system  that  simulates  fire  fighting  on 
board  a  Royal  Netherlands  Navy  frigate.  In  this  system,  the  trainee  assumes  the  role  of  the  commanding 
officer,  the  Officer  of  the  Watch  (OW).  To  perform  his  tasks,  the  OW  communicates  with  four  other  officers, 
all  of  which  are  simulated  by  intelligent  agents  developed  using  the  Belief -Desire-Intention  (BDI)  framework. 
The  DA  is  also  modelled  as  a  BDI  agent  but  operates  “behind  the  scenes”  based  on  information  from  the 
virtual  simulation  and  the  trainee.  The  DA  affects  training  through  control  of  the  simulated  environment 
and/or  the  behaviour  of  the  other  agents.  The  authors  concluded  that  their  project  demonstrated  that  the  DA 
provided  value  added  for  embedded  virtual  simulations. 

7.2  Mind  mapping  Exercises  for  Session  4 

Dr.  Robert  Sottilare  (USA)  and  Ms.  Claudy  Koehuis  (NED)  acted  as  facilitators  for  the  fourth  mind  mapping 
exercise,  which  concerned  issues  related  to  learning  in  EVS.  The  discussion  was  centred  on  the  following  two 
issues: 

7.2.1  What  are  the  factors  that  enable  or  limit  learning  in  EVS  training? 

Some  of  the  key  factors  include 

•  availability  of  instructors  and  role  players 

•  noises  and  other  distractions  inherent  in  the  operational  environment 

•  whether  trainees  possess  appropriate  metacognitive/learning  strategies  to  support  training 

•  ability  of  training  strategy  to  adapt  to  individual  learning  needs 

•  ability  of  training  strategy  to  adapt  to  individual  stress  levels  or  affective  states 

•  competition  of  training  with  operational  requirements. 

7.2.2  How  is  feedback  provided  to  trainees  during  embedded  training? 

8.0  CONCLUSIONS 

Some  of  the  most  mature  applications  of  EVS  have  been  in  command  and  control  (C2)  domains.  Examples  are 
command  centres  for  missile  defence  and  shipboard  AEGIS  systems.  The  advantages  of  such  domains  are  that 
EVS  systems  are  not  subject  to  severe  weight  and  space  limitations,  and  that  computational  resources  are 
relatively  plentiful.  Also,  typical  C2  displays  do  not  require  high-fidelity  visual  systems  to  create  realistic 
virtual  environments. 
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On  the  other  hand,  developments  in  visual  simulation  have  led  to  significant  advances  in  EVS  for  ground  and 
air  combat  vehicle  platforms.  These  advances  are  typified  in  the  development  of  the  FCS  and  the  F-35,  both  of 
which  explicitly  required  embedded  training  to  be  included  in  their  overall  design  concepts.  Both  systems 
have  been  depicted  as  archetypes  for  the  application  of  embedded  training  systems  in  new  combat  vehicles. 

Training  for  dismounted  warfighters  remains  perhaps  the  most  difficult  application  for  embedded  training. 
This  is  partly  due  to  the  severity  of  size  and  weight  restrictions  for  man-carried  systems.  Also,  simulation 
and/or  augmentation  of  unaided  vision  remain  significant  challenges.  Although  prototype  embedded  training 
systems  for  dismounted  warfighters  exist,  limitations  such  as  these  have  seriously  restricted  their  capabilities. 

A  fundamental  requirement  for  EVS  systems  intended  for  deployed  personnel  and  crews  is  they  minimize  the 
need  for  external  instructional  support.  The  emerging  technologies  of  intelligent  agents  and  intelligent  tutoring 
systems  provide  the  capability  to  embed  instructional  support  into  the  training  system.  However,  the 
technologies  are  relatively  immature  and  progress  has  been  limited  to  demonstration  projects. 

Although  the  effectiveness  of  EVS  was  documented,  conference  members  expressed  disappointment  that  the 
technology  has  not  been  more  generally  accepted  by  system  and  training  developers.  One  inhibiting  factor  that 
was  identified  in  mind  mapping  exercises  was  the  lack  of  NATO  guidance  on  development  of  EVS  systems.  It 
was  suggested  that  the  present  NATO  group  continue  to  monitor  the  evolution  of  EVS  technology  to  develop 
appropriate  guidance  and  policy. 
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